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(57) ABSTRACT 

A graphical user interface is proposed for generating an 
interface signal with a target on a display aimed. An image 
sensor has an image plane on which an image of the display 
including characteristic points is formed, a point of the 
image formed at a predetermined position of the image plane 
corresponding to the target. A processor calculates the 
position of the target on the display relative to the charac- 
teristic points on the basis of the output of the image sensor 
including the information of the positions of the character- 
istic points. And the interface signal is generated with the 
position of the target calculated. More specifically, the 
processor includes an image processor for identifying the 
positions of the characteristic points on the image plane, a 
first processor for calculating a first data on the basis of the 
positions of the characteristic points on the image plane, and 
a second processor for calculating a second data on the basis 
of the first data and the positions of the characteristic points 
on the image plane. This is not only useful in the graphical 
user interface, but also in detecting the position of a target 
point or the attitude of a three-dimensional object in general. 

31 Claims, 25 Drawing Sheets 
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GRAPHICAL USER INTERFACE AND positions of the characteristic points. And the interface 

POSITION OR ATTITUDE DETECTOR signal is generated with the position of the target calculated. 

Thus, the interface signal is generated at any desired place 

remote from the display. 

BACKGROUND OF THE INVENTION 5 According to another feature of the invention, aiming 

| . . • < j , , . . f device is provided for helping the image sensor to form the 

This application is based upon and claims priority of . f f, , .. u \ u * . . a . .« „ - t . t . 

Japanese Patent Applications No 11-252732 filed on Sep. 7, ma f, of the ^ wth the ,ar S et m ^incidence with the 

1999, No. 11-281462 filed on Oct. 1, 1999, No. 2000- P^ermued ponton. 

218969 filed on Jul. 19, 2000 and No. 2000-218970 filed on in According to still another feature of the invention a 

Jul. 19, 2000, the contents being incorporated herein by 10 controller orders the image sensor to take the image of the 

reference display. More specifically, the controller is designed to order 

v m f ih t ' ^ e * ma S e sensor to take one image in response to a manual 

1. Meld ot tne Invention operation Alternatively, the controller is designed to order 
The present invention relates to a Graphical User Inter- the image sensor t0 take a series of i^ges j n succession. 

face (GUI) for a computer or an amusement game, and to a is Accordi to a feature of the inventioD) t he 

detector for detecting a position or an attitude of a target. processor for calculating the position of the target includes 

2. Description of Related Art a n image processor for identifying the positions of the 
In this field of the art, various types of Graphical User characteristic points on the image plane. The processor 

Interface have been proposed for a computer or an amuse- further includes a first processor for calculating a first data 

ment game. 20 on the basis of the positions of the characteristic points on 

For example, a Graphical User Interface on a wide screen the image plane. Finally the processor includes a second 

display has been proposed for the purpose of a presentation processor for calculating a second data on the basis of the 

or the like, in which a projector is connected to a computer first data and the positions of the characteristic points on the 

for projecting a display image on a wide screen so that a 25 image plane. The position of the target on the display is 

number of audience may easily appreciate the presentation. detected on the basis of the second data. 

In this case, a laser pointer or the like is prepared for The above mentioned feature is not only useful in Graphi- 

pointing an object such as an icon or a text on the wide cal User Interface, but also in detecting the position of a 

screen to input a command relating to the object pointed by target point or the attitude of a three-dimensional object in 

the laser pointer. 30 general. 

In an amusement such as a shooting game, on the other In the invention, the information of attitude of the object 

hand, a scene including an object is presented on a cathode is utilized by the position detection, and then by Graphical 

ray tube display of a game machine under a control of a User Interface. In terms of the position detection or Graphi- 

computer. A user remote from the display tries to aim and cal User Interface, the target point should be aimed for 

shoot the object with a gun, and the game machine judges 35 detection. However, any specific target point need not be 

whether or not the object is successfully shot. aimed in the case of solely detecting the attitude of an object 

For the purpose of the above mentioned Graphical User as lon S as the of necessary characteristic points of the 

Interface, it has been proposed to fixedly locate a CCD m formed on the image sensing plane, 

camera relative to the screen for detecting a bright spot on According to another feature of the invention, the output 

the wide screen caused by a laser beam to thereby detect the 40 of the image sensor is transmitted to an outside processor for 

position of the bright spot on the screen. a calculation of the position of the target. This is advanta- 

It has been also proposed to prepare a plurality of light S eous m utilizin g graphical user interface in a computer or 

emitting elements on the screen in the vicinity of the a shooting game. 

projected image for the purpose of analyzing at a desired According to a further feature of the invention, a projector 

place the intensities and directions of light received from the 45 projects an image including characteristic points on a screen 

light emitting elements to detect a position of a target on the acting as the display, and the image on the screen is received 

screen. DV the image sensor. In this case, the position sensing is 

However, there have been considerable problems and successful even if an optical axis of the projector is inclined 

disadvantages still left in the related art, such as in seeking Wlth res P ect to tne screen * 

freedom or easiness in use, accuracy or speed in detection 50 0ther features and advantages according to the invention 

and size or cost in the practical product. w iU be readily understood from the detailed description of 

Laid-open Patent Application Nos. 2-306294, 3-176718, the P refe ^ red embodiments in conjunction with the accom- 

4-305687, 6-308879, 9-231373 and 10-116341 disclose panying drawings. 

various attempts in this field of art. $5 BRIEF DESCRIPTION OF THE DRAWINGS 

SUMMARY OF THE INVENTION FIG. 1 represents a perspective view of Embodiment 1 

according to the present invention. 

In order to overcome the problems and disadvantages, the nG 2 XDts a block diagrams of tne main bod of 

invention provides a graphical user interface for generating Embodiment 1 

an interface signal with a target on a display aimed. In the 60 _ T _ . ' . 4 ., , 4 . , . , , f 

graphical user interface, an image sensor has an image plane 3 ^presents a detailed partial block diagram of FIG. 
on which an image of the display including characteristic 

points is formed, a point of the image formed at a prede- FIG - 4 represents the perspective view of the main body, 

termined position of the image plane corresponding to the FIG. 5 represents a cross sectional view of the optical 

target. A processor calculates the position of the target on the 65 system of main body in FIG. 4. 

display relative to the characteristic points on the basis of the FIG. 6 represents a cross sectional view of a modification 

output of the image sensor including the information of the of the optical system. 
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FIG. 7 represents a flow chart for the basic function of Main body 100 may be handled at any desired position 

Embodiment 1 according to the present invention. relative to plane 110. Broken line 101 is an optical axis of 

FIG. 8 represents a flow chart of the manner of calculating image sensing plane of camera 1 located within main body 

the coordinate of the target point and corresponds to the 100, the optical axis leading form the center of the image 

details of step 106 in FIG. 7. 5 sensing plane perpendicularly thereto to target point Ps on 

FIG. 9 represents an image taken by the main body, in P^ ane 

which the image of target point is within the rectangular ^ P lane t0 be detected b V Embodiment 1 is a rectangle 

shape defined by the four characteristic points. appearing on an object or a figure such as a display of a 

™~ 1A . r . * i u *u monitor for a personal computer, a projected image on a 

FIG. 10 represents another type of image taken by the ... y t u- • TU u ♦ • 

• . ■. • !_• L 4 i_ • ; * • . • i ' j 4 u screen, or a computer graphic image. The characteristic 

mam body, in which the image target point is outside the . j- * * ■ *• u *u 

, , -.^jirLr L i • t points according to the present mvention may be the corners 

rectangular shape defined by the four characteristic points. f, . - ** . r . . ■ ♦ j 

& v J r themselves of a rectangular image projected on a screen. 

FIG. 11 represents an image under the coordinate con- Alternatively, the characteristic points may be projected 

version from X-Y coordinate to X'-Y' coordinate. witMn an on the SCfeen t0 define a new rectangular 

FIG. 12 represents a two-dimensional graph for explain- is inside the image projected on the screen, 

ing the basic relationship among various planes in the FIG. 2 shows the block diagrams of the main body 100, 

perspective projection conversion. wnile Fia 3 a detailed partial block diagram of FIG. 2. 

FIG. 13 represents a two-dimensional graph of only the Further, FIG. 4 represents the perspective view of main body 

equivalent image sensing plane and the equivalent rectan- 100. 

gular plane with respect to the eye. 20 In pio. 2, camera 1 includes a lens system and an image 

FIG. 14 represents a three-dimensional graph for explain- sensor. Camera 1 may be a digital still camera with CCD, or 

ing the spatial relationship between X-Y-Z coordinate rep- a video camera. 

resenting the equivalent image sensing plane in a space and Camera 1 needs to define an aiming point for designating 

X*-Y* coordinate representing the given rectangular plane. ^ the target point Ps on plane 110. According to Embodiment 

FIG. 15 represents a three-dimensional graph showing a 1, the aiming point is defined at the center of the image 

half of the given rectangular plane with characteristic points sensing plane as origin Om of image coordinate (X-Y 

Ql and Q2. coordinate). A/D converter 2 converts the image data taken 

FIG. 16 represents a two-dimensional graph of an b y camera 1 into digital image data. Frame memory 3 

orthogonal projection of the three-dimensional rectangular 30 temporally stores the digital image data at a plurality of 

plane in FIG 15 onto X'-Z' plane. addresses corresponding to the location of the pixels of 

FIG. 17 represents a two-dimensional graph of a orthogo- CCT " * raD Jf, c me f m °? 3 is °f a ° f ^ ^ 

nal projection of the three-dimensional rectangular plane in ^gabytes (MS) for storing a plurality of images which will 

FIG 15 onto Y'-Z' plane. be taken successively. 

A t_r ¥ TX7 j * . «. « « Controller 4 includes Read Only Memory (ROM) storing 

FIG. 18 A represents a graph of U-V coordinate in which 35 . ■ *u *• • i i 

y ,. A & , * A . . 4 . A a program for processing the perspective view calculation 

a pomt corresponding to characteristic point Q3 is set as *, / . fi_ • a t 

ori n O a a P ro S ram ^ or controlling other various functions. Image 

on & a ' processor 5 includes characteristic point detector 51 and 

FIG. 18B represents a graph of X*-Y* coordinate in position calculator 52. Characteristic point detector 51 

which Om is set as the origin. ^ detects the characteristic points defining the rectangular 

FIG. 19 represents a perspective view of Embodiment 2 plane in a space on the basis of the image data taken by 

according to the present invention. camera 1, the detailed structure of characteristic point detec- 

FIG. 20 represents a block diagram of the system concept tor 51 being explained later. Position calculator 52 deter- 

for main body of Embodiment 2. mines the position of the target point on the basis of the 

FIG. 21 represents a perspective view of the main body of 45 coordinate of the identified characteristic points. 

Embodiment 2. Although not shown in the drawings, characteristic point 

t-t¥/-i . U1 i A - , i . detector 51 includes a detection checker for checking 

FIG. 22 represents a block diagram ot personal computer . , . . . & 

to which the image data or command execution signal is ^f th f r or n °' the characteristic points have been success- 
transferred from the main body. MX J detected from J° e ^ ima S? ^ta tempora ly stored 

n ^ n , , - , . so id frame memory 3 under the control of controller 4. By 

FIG. 23 represents a flow chart showing the operations of means of such a checker> an tQr who has ^ t0 take 

Embodiment 2 according to the present invention. aQ jnu& data sufficieQt tQ detect ^ characteristic points 

FIG. 24 represents a flow chart showing the operation of can be warned by a sound or the like to take a new image 

Embodiment 2 in the one-shot mode. again. 

FIG. 25 represents a flow chart showing the operation of 55 Position calculator 52 includes attitude calculator 521 for 

Embodiment 2 in the continuous mode. calculating the rotational parameters of the given rectangular 

plane in a space (defined by X-Y-Z coordinate) relative to 

DETAILED DESCRIPTION OF THE the image se^g p i ane coordinate calculator 522 for 

PREFERRED EMBODIMENTS calculating the coordinate of the target point on the rectan- 

Embodimentl 60 plane. 

FIG, 3 represents a detailed block diagram of attitude 

FIG. 1 represents a perspective view of Embodiment 1 calculator 521, which includes vanishing point processor 

showing the system concept of the position detecting device 523, coordinate converter 5214 and perspective projection 

according to the present invention. converter 5215. Vanishing point processor 523 includes 

Main body 100 of Embodiment 1 is for detecting a 65 vanishing point calculator 5211, vanishing line calculator 

coordinate of a target point Ps on a given rectangular plane 5212 and vanishing characteristic point calculator 5213 for 

110 defined by characteristic points Ql, Q2, Q3 and Q4. finally getting the vanishing characteristic points on the 
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basis of the vanishing points calculated on the coordinate of termined point on the image sensing plane of CCD 11. 

the plurality of characteristic points on the image sensing Human eye 70 observes both cross 74 and the image of 

plane, Perspective projection converter 5215 is for finally rectangular plane 110 on focal plane 73 by way of eyepiece 

calculating the rotational parameters. 71 and mirror 72. Thus, if the image is sensed by camera 1 

In FIG. 4, light beam emitter 6A made of semiconductor 5 with cross 74 located at the image of the target point on focal 

laser is a source of light beam to be transmitted toward the plane 73, the target point is sensed at the predetermined 

rectangular plane for visually pointing the target point on the point on the image sensing plane of CCD 11. 
plane such as in a conventional laser pointer used in a 

presentation or a meeting. As an alternative of light beam If octangular plane 110 is an image projected by a video 

emitter 6A, a light emitting diode is available. 10 projector controlled by a computer, for example, light beam 

FIG. 5 shows a cross sectional view of the optical system cmitter 6Aor °P tical finder 6B ma y be omitted - ^ this case, 

of main body 100 in FIG. 4. If a power switch is made on, the 0Ut P ut si &* 1 * forwarded from the output signal pro- 

the laser beam is emitted at light source point 60 and cessor 9 to the computer, and the calculated position is 

collimated by collimator 61 to advance on the optical axis of displayed on the screen as a cursor or the like under the 

camera 1 toward rectangular plane 110 by way of mirror 62 15 contro1 of the computer. Thus, the point which has been 

and semitransparent mirror 13A. Camera 1 includes objec- at ^ he predetermined point of the image sensing plane 

tive lens 12 and CCD 11 for sensing image through semi- of CCD 11 15 fed back t0 a user who 15 viewin g the screen - 

transparent mirror 13A, the power switch of the laser being Therefore, if the image taking, the calculation and the feed 

made off when the image is sensed by camera 1. Therefore, back displaying functions will be repeated successively, the 

mirror 13A may alternatively be a full refractive mirror, 20 user can finall y locate the at the tar § et P omt on the 

which is retractable from the optical axis when the image is screen. 

sensed by camera 1. Similarly, in the case that the given rectangular plane is a 

In this Embodiment 1, the position of beam emitter 6A is monitor of a personal computer, user can control the cursor 

predetermined relatively to camera 1 so that the path of laser on the monitor from any place remote form the computer by 

beam from beam emitter 6A coincides with the optical axis means of the system according to the present invention. 

of camera 1. By this arrangement, a point on the rectangular T i- u j - * i u i j- e 

plane 110 which is lit by the laser beam coincides with a In Embodiment 1 above camera 1 and .mage processor 5 

j. - j «. i , u , + . . are integrated as a one body as in FIG. 4. Image processor 

predetermined pomt, such as the center, on the image , ^ L A ' c _ * j t5 i * j 

i Jppn 11 Tn. *u • - au 5 may, however, be separated from camera 1, and be located 

sensing plane of CCD 11. Thus, if the image is sensed by J ' «. *• ■ *u t • u * a - 

6 [ . 4LlL , , j * *i_ * * • * *u 30 as a software function in the memory of a peripheral device 

camera 1 with the laser beam aimed at the target point, the ^ J r r 

target point is sensed at the predetermined point on the " 

image sensing plane of CCD 11. The laser beam is only help FIG. 7 represents a flow chart for the basic function of 

for this purpose. Therefore, the position of light beam Embodiment 1 according to the present invention. In step 

emitter 6A relative to camera 1 may alternatively predeter- S100, the main power of the system is turned on. In step 

mined so that the path of laser beam from beam emitter 6A S101, the target point on a given plane having the plurality 

runs in parallel with the optical axis of camera 1 with mirrors of characteristic points is aimed so that the target point is 

62 and 13A removed. In this case, the difference between the sensed at the predetermined point on the image sensing 

path of laser beam and the optical axis of camera 1 can be plane of CCD 11. According to Embodiment 1, the prede- 

corrected in the course of calculation. Or, the difference may termined point is specifically the center of image sensing 

be in some case negligible. plane of CCD 11 at which the optical axis of the objective 

In FIG. 2, power switch 7 for light beam emitter 6A and lens of camera 1 intersects, 

shutter release switch 8 are controlled by a dual step button , n 1Q2 ^ ^ {& ^ {n se tQ 

in which power switch 7 is turned on with a depression of switcfa g of ^ camera ± ^ me ^ of ^ mt 

the dual step button to the first step^ If the depression is 45 at ^ edetermined point on the ^ sensing lane of 

quitted at the first step power switch 7 is simply turned off CCD u tfaen ^ { s{ { b&[ st0fed in ^ framc 

On the contrary, if the dual step button is further depressed me b of necessary si al p rocess i ag following 

to the second step, shutter release switch 8 is turned on to ±t { ^ 
sense the image, the power switch 7 being turned off in the 

second step of the dual step button. 5Q In step S103, the characteristic points defining the rect- 
Output signal processor 9 converts the attitude data or the angular plane are identified, each of the characteristic points 
coordinate data calculated by position calculator 52 into bein S tbc center of of cach of predetermined marks, 
output signal for displaying the output signal as a numeral respectively. The characteristic points are represented by 
data with the taken image on the main body or for forward- coordinate ql, q2, q3 and q4 on the basis of image sensing 
ing the output signal to the peripheral apparatus, such as a 55 P lane coordinate. In step 104, it is tested whether the desired 
video projector or a computer. By means of transmitting the four characteristic points are successfully and accurately 
output signal with a wireless signal transmitter adopted in identified. If the answer is "No", a warning sound is gen- 
output signal processor 9, the system will be still more erated in ste P S105 to prompt the user to take an image 
conveniently used. again. On the other hand, if the answer is "Yes", the flow 

In FIG. 6, optical finder 6B is shown, which can replace 60 advances to ste P S106 

light bean emitter 6A for the purpose of aiming the target Step S106 is for processing the rotational parameters of 

point so that the target point is sensed at the predetermined the given rectangular plane in a space relative to the image 

point on the image sensing plane of CCD 11, sensing plane and the coordinate of the target point on the 

Optical finder 6B includes focal plane 73 which is made rectangular plane, which will be explained later in detail. In 

optically equivalent to the image sensing plane of CCD 11 65 step S107, the calculated data is converted into output signal 

by means of half mirror 13B. Cross 74 is positioned on focal for display (not shown) or transmission to the peripheral 

plane 73, cross 74 being optically equivalent to the prede- apparatus. Then, the flow will be ended in step S108. 
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Now, the description will be advanced to the detailed Referring to FIG. 8, linear equations for lines qlq2, q2q3, 

functions of image processor 5 of Embodiment 1. q3q4 and q4ql are calculated on the basis of coordinates for 

(A) Characteristic Point Detection detected characteristic points ql, q2, q3 and q4 in step Sill, 
Various types of characteristic point detector are possible lines qlq2, q2q3, q3q4 and q4ql being defined between 

according to the present invention. 5 neighboring pairs among characteristic points ql, q2, q3 and 

For example, in the case that the given rectangular plane q4, respectively. In step S112, vanishing points TO and SO 

is an image projected on a screen by a projector, the are calculated on the basis of the liner equations. Steps 111 

characteristic points are the four corners Ql, Q2, Q3 and Q4 and 112 correspond to the function of vanishing point 

of a rectangular image projected on a screen as in FIG. 1. calculator 5211 of the block diagram in FIG. 3. 

The image is to taken with all the four corners covered The vanishing points defined above exists in the image 

within the image sensing plain of the camera. For the without fail if a rectangular plane is taken by a camera. The 

purpose of detecting the corners without fail in various vanishing point is a converging point of lines. If lines qlq2 

situations, the projector is arranged to alternately projects a and q3q4 are completely parallel with each other, the van- 

bright and dark images and the camera is released twice in ishing point exists in infinity. 

synchronism with the alternation to take the bright and dark According to Embodiment 1, the plane located in a space 

images Thus, the corners are detected by the difference 15 is a rectangular having two pairs of parallel lines, which 

between the bright and dark images to finally get the binary cause two vanishing points on the image sensing plane, one 

picture. According to Embodiment 1 in FIG. 2, characteristic vanishing point approximately on the direction along the 

point detector 51 includes difference calculator 511, binary X-axis, and the other along the Y-axis, 

picture processor 512 and characteristic point coordinate In FIG. 9, the vanishing point approximately on the 

identifier 513 for this purpose. 20 direction along the X-axis is denoted with SO, and the other 

Alternatively, at least four marks may be projected within along the Y-axis with TO. Vanishing point TO is an intersec- 

an image on the screen to define a new rectangular inside the tion of lines qlq2 and q3q4. 

image projected on the screen, each of the characteristic In step S113, linear vanishing fines OmSO and OmTO, 

points being calculated as the center of gravity of each of which are defined between vanishing points and origin Om, 

marks. Also in this case, the projector is arranged to alter- 25 are calculated. This function in Step 113 corresponds to the 

nately projects two images with and without the marks, and function of vanishing line calculator 5212 of the block 

the camera is released twice in synchronism with the alter- diagram in FIG. 3. 

nation to take the two images. Thus, the marks are detected Further in step S113, vanishing characteristic points qsl, 

by the difference between the two images to finally get the qs2, qtl and qt2, which are intersections between vanishing 

binary picture. 30 lines OmSO and OmTO and lines q3q4, qlq2, q4ql and 

The characteristic points may be detected by an edge q2q3, respectively, are calculated. This function in Step 113 

detection method or a pattern matching method. In the corresponds to the function of vanishing characteristic point 

pattern matching method, the reference image data may calculator 5213 of the block diagram in FIG. 3. 

previously stored in memory of the system to be compared The coordinates of the vanishing characteristic points are 

with a taken image. 35 denoted with qsl(Xsl,Ysl), qs2(Xs2,Ys2), qtl(Xtl,Ytl) 

(B) Position Calculation and qt2(Xt2,Yt2). Line qtlqt2 and qslqs2 defined between 
Position calculator calculates a coordinate of a target the vanishing characteristic points, respectively, will be 

point Ps on a given rectangular plane defined by character- called vanishing lines as well as OmSO and OmTO. 

istic points, the given rectangular plane being located in a Vanishing lines qtlqt2 and qslqs2 are necessary to cal- 

space. 40 culate target point Ps on the given rectangular plane. In other 

FIG. 8 show the manner of calculating the coordinate of words, vanishing characteristic points qtl, qt2, qsl and qs2 

the target point and corresponds to the details of step 106 in on the image coordinate (X-Y coordinate) correspond to 

FIG. 7. points Tl, T2, SI and S2 on the plane coordinate (X*-Y* 

FIGS. 9 and 10 represent two types of image q taken by coordinate) in FIG. 1, respectively, 

main body 100 from different position relative to the rect- 45 If the vanishing point is detected in infinity along X-axis 

angular plane, respectively. In FIGS. 9 and 10, the image of of the image coordinate in step S112, the vanishing line is 

target point Ps is in coincidence with predetermined point considered to be in parallel with X-axis. 

Om, which is the origin of the image coordinate. Charac- In step 114, image coordinate (X-Y coordinate) is con- 

teristic points ql, q2, q3 and q4 are the images on the image verted into X'-Y' coordinate by rotating the coordinate by 

sensing plane of the original of characteristic points Ql, Q2, 50 angle P around origin Om so that X-axis coincides with 

Q3 and Q4 on the rectangular plane represented by X*-Y* vanishing line OmSO. Alternatively, image coordinate (X-Y 

coordinate. coordinate) may be converted into X"-Y" coordinate by 

In FIG. 9, the image of target point at predeternmined rotating the coordinate by angle a around origin Om so that 

point Om is within the rectangular shape defined by the four Y-axis coincides with vanishing line OmTO. Only one of the 

characteristic points ql, q2, q3 and q4, while the image of 55 coordinate conversions is necessary according to Embodi- 

target point at predetermined point Om is outside the rect- ment 1. (Step S114 corresponds to the function of coordinate 

angular shape defined by the four characteristic points ql, converter 5214 in FIG. 3.) 

q2, q3 and q4 in FIG. 10. FIG. 11 is to explain the coordinate conversion from X-Y 

(bl) Attitude Calculation coordinate to X'-Y' coordinate by rotation by angle p around 

Now, the attitude calculation, which is the first step of 60 origin Om with the clockwise direction is positive. FIG. U 

position calculation, is to be explained in conjugation with also explains the alternative case of coordinate conversion 

the flow chart in FIG. 8, the block diagram in FIG. 3 and from X-Y coordinate to X"-Y f ' coordinate by rotating the 

image graphs in FIGS, 9 to 11. The parameters for defining coordinate by angle a, 

the attitude of the given plane with respect to the image The coordinate conversion corresponds to a rotation 

sensing plane are rotation angle y around X-axis, rotation 65 around Z-axis of a space (X-Y-Z coordinate) to determine 

angle around Y-axis, and rotation angle a or p around one of the parameters defining the attitude of the given 

Z-axis. rectangular in the space. 
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By means of the coincidence of vanishing line qslqs2 to as "plane coordinate") representing the given rectangular 

with X-axis, lines Q1Q2 and Q3Q4 are made in parallel with plane. Z-axis of image coordinate intersects the center of the 

X-axis. equivalent image sensing plain perpendicularly thereto and 

In step S115, characteristic points ql, q2, q3 and q4 and coincides with the optical axis of the objective lens. View 

vanishing characteristic points qtl, qt2, qt3 and qt4 on the 5 point O for the perspective projection conversion is on 

new image coordinate (X'-Y' coordinate) are related to Z " axis a P art from origin 0m of the ima S e coordinate by f 

characteristic points Ql, Q2, 03 and Q4 and points Tl, T2, R ota f ion ™& y around X-axis, rotation angle $ around 

SI and S2 on the plane coordinate (X*-Y* coordinate). This Y " a ^ an ^ two rotation angles a and P both around Z-axis 

is performed by perspective projection conversion according are defined with respect o the image coordinate, the clock- 

* .l / n i *u *• ■ *■ wise direction being positive for all the rotation angles. With 

to the geometry. By means of the perspec ive projection 10 % ^ ^ ^ coordinate % ^ for 

conversion^ the attitude of the given rectangular plane in the £ tive proje ction conversion, Ze-axis being coincident 

space (X-Y-Z coordinate) on the basis of the image sensing with and Xe . axis and Ye . axis ^ {n M with 

plane is calculated. In other words, the pair of parameters, which win x _ axis and y-axis, respectively, 

angle ((> around Y-axis and v around X-axis for defining the Now the perspective projection conversion will be 

attitude of the given rectangular plane are calculated. The is described in detail. According to the geometry on FIG. 13, 

perspective projection conversion will be discussed in detail me relationship between Ye-Ze coordinate of a point such as 

in the following subsection (bll). (Step S115 corresponds to Ql on the equivalent rectangular plane and that of a point 

the function of perspective projection converter 5215 in such as ql on the equivalent image sensing plane, the points 

FIG. 3.) Ql and ql being viewed in just the same direction from view 

In step S116, the coordinate of target point Ps on the plane 20 point O, can be generally expressed by the following equa- 

coordinate (X*-Y* coordinate) is calculated on the basis of tions (1) and (2) 
the parameters gotten in step S115. The details of the 

calculation to get the coordinate of target point Ps will be Yf (1) 

discussed later in section (b2). ~ f -Ytany 

(bll) Perspective Projection Conversion 25 

Perspective projection conversion is for calculating the v _ f ® 

parameters (angles <j> and angle y) for defining the attitude of f~ Y ***** 
the given rectangular plane relative to the image sensing 

plane on the basis of the four characteristic points identified Therefore, characteristic points Qi(Y*i,Z*i), wherein i is 

on image coordinate (X-Y coordinate). 30 

FIG. 12 explains the basic relationship among various 
planes in the perspective projection conversion, the relation- 
ship being shown in two-dimensional manner for the pur- \ y f f 2 1 (3) 

pose of simplification. According to FIG. 12, a real image of Q\ (>7, 7\) = 1 — , j-t 

the given rectangular plane is formed on the image sensing 35 [J J i 7\ 

plane by the objective lens of the camera. The equivalent r Yl -f f 2 } (4) 

image sensing plane denoted by a chain line is shown on the <?2(>2. ^) = L_ K2tanr » / - y 2 taar J 

object side of the objective lens at the same distance from 

the objective lens as that of the image sensing plane form the 

objective lens, the origin Om and points ql and q2 being also 40 FIG. 14 shows the perspective projection conversion in 

used in the X-Y coordinate of the equivalent image sensing three -dimensional manner for calculating the attitude of the 

plane denoted by the chain line. The equivalent rectangular rectangular plane given in a space (X-Y-Z coordinate) 

plane denoted by a chain line is also set by shifting the given relative to the image sensing plane. Hereinafter the equiva- 

rectangular plane toward the object side so that target point lent image sensing plane and the equivalent rectangular 

Ps coincides with the origin Om with the equivalent rect- 45 plane will be simply referred to as "image sensing plane" 

angular plane kept in parallel with the given rectangular and "given rectangular plane", respectively, 

plane. The points Ql and Q2 are also used in the equivalent The given rectangular plane is rotated around Z-axis, 

rectangular plane. In this manner, the relationship between which is equal to Z'-axis, by angle p in FIG. 14 so that 

the image sensing plane and the given rectangular plane are Y'-axis is made in parallel with Ye-axis not shown, 

viewed at origin Om of the equivalent image sensing plane 50 In FIG. 15, a half of the given rectangular plane is shown 

as if viewed from the center O of the objective lens, which with characteristic points Q1(X*1, Y*l, Z*l) and Q2(X*2, 

is the view point of the perspective projection conversion. Y*2, Z*2). Points Tl(X*tl, Y*tl, Z*tl) T2(X*t2, Y*t2, 

FIG. 13 shows only the equivalent image sensing plane Z*t2) and S2(X*s2, Y*s2, Z*s2) are also shown in FIG. 15. 

and the equivalent rectangular plane with respect to view The remaining half of the given rectangular plane and the 

point O. The relationship is shown on the basis of Ye-Ze 55 points such as Q3, Q4 and SI are omitted from FIG. 15. 

coordinate with its origin defined at view point O, in which Further, there are shown in FIG. 15 origin Om(0,0,f) coin- 

the equivalent image sensing plane on X-Y coordinate and cident with target point Ps and view point 0(0,0,0), which is 

the given rectangular plane on X*-Y* coordinate are shown, the origin of Xe- Ye-Ze coordinate. 

Z-axis of the equivalent image sensing plane being in Line TlOm is on Y'-Z' plane and rotated by angel y 

coincidence with Ze-axis. View point O is apart from origin 60 around X'-axis, while line S20m is on X'-Z' plane and 

Om of the image coordinate by f, which is equal to the rotated by angel $ around Y' -axis, the clockwise directions 

distance from the objective lens to the image sensing plane. of rotation being positive, respectively. The coordinates of 

Further, the given rectangular plane is inclined by angle y. Ql, Q2, Tl, T2 and S2 can be calculated on the basis of the 

FIG. 14 is an explanation of the spatial relationship coordinates of ql, q2, qtl, qt2 and qs2 through the perspec- 

between X-Y-Z coordinate (hereinafter referred to as "image 65 live projection conversion. 

coordinate'*) representing the equivalent image sensing FIG. 16 represents a two-dimensional graph showing an 

plane in a space and X*-Y* coordinate (hereinafter referred orthogonal projection of the three-dimensional rectangular 
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plane in FIG. 15 onto X*-Z' plane in which Y'-O. In FIG. 16, 

only line S1S2 denoted by the thick line is really on X'-Z* f / 2 ta# ( 15 ) 

plane, while the other lines on the rectangular plane are on /-K/^-tany = f-tan^-x;, 

the X'-Z' plane through the orthogonal projection. 

According to FIG. 16, the X'-Z' coordinates of Tl(X*tl, 5 
Z*tl), T2(X*t2,Z*t2), Sl(X*sl,Z*sl), S2(X*s2,Z*s2) and Similarly, the following equation (16) results from equa- 
Q1(X*1,Z*1) can be geometrically calculated on the basis tions (9) and (14) both relating to the Z*-coordinate of Ql: 
of the X'-Z' coordinates of qtl(X'tl,f), qt2(X't2,f), qsl(X'sl, 

f), qs2(X's2,f) and ql(X'l,f) and angle 4 as in the following / . r = _L, . r < i6) 

equations (5) to (9): 10 /-K/ taay 32 X^ * /-ta# - X[ * 

Ti{X'f, z;,) = [^/■/■tan^^ / 2 -ta# 1 (5) Equation (15) can be simplified into the following equa- 

[/•ta#-x,; /.tamfr-^J tion (17): 

ta# Y' t! 



X s '/ • tan^ 4- / ' X,'/ ■ tan^fr + / 



And equation (15) can be modified into the following 

r f 2 1 *® 20 c q uat ion (18) by substituting X*s2 and Z*s2 with equation 

5 2 (^, = [x^^Tj' x^-tan^ + /J (8), and tan y with equation (17) 

For the purpose of the following discussion, only one of Equations (17) and (18) are conclusion of defining angles 
X'-Z' coordinates of the characteristic points Ql to Q4 is y and ^ which are the other ^ 0 of parameters for defining 
necessary. Equation (9) for Ql may be replaced by a similar the att i tu de of the given rectangular plane relative to the 

equation for any one of characteristic points Q2 to Q4. 30 image sensing plane. The value for tan y given by equation 
On the other hand, FIG. 17 represents a orthogonal (17) can b e practically calculated by replacing tan $ by the 
projection of the three-dimensional rectangular plane onto va i ue calculated through equation (18). Thus, all of the three 
Y'-Z' plane in which X'=0. In FIG. 17, only line T1T2 angles (3, y and $ are obtainable. 

denoted by the thick line is really on Y'-Z* plane, while the M in equat ions (17) and (18), angles y and y are expressed 

other lines on the rectangular plane are on the Y'-Z' plane 35 by the coordinate of characteristic point ql(X'l, Y'l) and the 
through the orthogonal projection. coordinate of a vanishing characteristic points qtl(X'tl, 

According to FIG. 17, the Y'-Z' coordinates of Tl(Y*tl, Y 'tl) and qs2(X's2) which are calculated on the basis of the 
Z*tl), T2(Y*t2,Z*t2), Sl(0,Z*sl), S2(0,Z*s2) and Q1(Y*1, coor dinates. Distance f in the equation is a known value. 
Z*l) can be geometrically calculated on the basis of the 7^ the at titude of the given rectangular plane relative to 

Y'-Z' coordinates of qtl(Y'tl,f), qt2(Y't2,f), qsl(Y'sl,f), 40 the scns ^ p i ane can t> e uniquely determined by the 

qs2(Y's2,f) and ql(Yl,f) and angle y as in the following positions of the characteristic points on the image plane, 
equations (5) to (9): According to present invention, any complex matrix 

conversion or the like is not necessary for calculating 
T\ (yj, z; t ) = \ Y ' tl 'f ^ 1 ^ parameters of the attitude of the given rectangular plane, but 

[/ - Ylitaoy f - K/ytanr J 45 suc ^ s i m pi e f orm 0 f equations as equations (17) and (18) are 
, 2 sufficient for the same purpose. This leads to various 

r 2 (>%,^)= - Y ' 2 / , -7 — y, advantages, such as a reduced burden on the calculating 

[/ - litany / - ^tanyj function, a less error or high accuracy in calculation and a 

_[ / 2 .tan r 1 (12) 5Q low cost of the product. 

<WVZw) = |0, y/ / + y >tair) ,J Further, only condition necessary for the calculation 

according to the present invention is that the characteristic 
_ r f 2 - tan? 1 (13) points on the given plane are required to define a rectangle. 

^2(^2. K2) - |°' Y; 2 + f-uwy\ In other words, any specific information such as the aspect 

ratio of the rectangle or the relation among the coordinates 
Gitr z*) - \ Y ' x r * zj (14J °£ tne corners °f me rectangle is not necessary at all . 

LZ-J'i -tairy * f-Y' r tany J Further, an information of the distance from the image 

sensing plane to the given plane is not necessary in the 
The Y*-coordinate of SI and S2 in equations (12) and (13) calculation according to the present invention, 

are zero since the X- Y coordinate is rotated around Z axis by 6 Q E 1 uations ( 19 ) and (2°) are another forms of conclusion, 
angle p s o that X-axis coincides with vanishing fine S1S2, m wmch the analysis is made with the counter clockwise 
angle p being one of the parameters for defining the attitude rotation around Y-axis defined as positive direction for 
of the given rectangular plane relative to the image sensing representing $ on the contrary to equations (17) and (18): 
plane. 

Since the Z*-coordinate of Tl in equation (5) is just the 65 tany = * — '— 

same as that in equation (10), the following equation (15) ^ Y>u 

results: 
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-continued 

V - V 



■f 



(20) 



14 



(25) 



In the case of equations (19) and (20),at least one coor- 
dinate of characteristic point ql(X*l, Y'l) at least one 
coordinate of a vanishing characteristic point qtl(X , tl > Y'tl) 
and distance f are only necessary to get angles v and <J>. 

Equations (21) and (22) are still another forms of 
conclusion, in which the analysis is made with the X-Y 
coordinate rotated around Z axis by angle a so that Y-axis 
coincides with vanishing line T1T2 on the contrary to 
equations (17) and (18): 



tan^ 



1 

tarry 



AS 



(21) 



tany = 



-XT 



(22) 



XhY\ - XZYZj 



_ O m Si 

OA 



\KA l^-tan^ + /| 



(23) 



\X; 2 \ IZ-tan^ + ^l 



10 



Alternatively, FIG. 17 also gives the ratio m and n 
expressed by Y'-coordinate of vanishing characteristic 
points qsl(Y*sl), qs2(Y ( s2), qtl(Y'tl) and qt2(Y't2), dis- 
tance f and angle y as in the following equations (26) and 
(27): 



O^l I^MKW-tanrl 



15 



O m T x _ |K,' f | |/-y f 2-tany| 



(26) 



(27) 



'\Y' t2 \ |/-nVtanrl 



20 



(b2) Coordinate Calculation 

Now, the coordinate calculation for determining the coor- 
dinate of the target point on the given rectangular plane is to 
be explained. The position of target point Ps on given 25 
rectangular plane 110 with the plane coordinate (X*-Y* 
coordinate) in FIG. 1 is calculated by coordinate calculator 
522 in FIG. 2 on the basis of the parameters for defining the 
attitude of the given rectangular plane obtained by attitude 
calculator 521. 

Referring to FIG. 16, ratio m=OmSl/OmS2 represents the 
position of Om along the direction in parallel with that of 
Q3Q2, while ratio n=OmTl/OmT2 represents the position 
of Om along the direction in parallel with that of Q3Q4, 
which is perpendicular to Q3Q2. And, ratio m and ratio n can 
be expressed as in the following equations (23) and (24), 
respectively, in view of equations (5) to (8) in which 
coordinates of Sl(X*sl,Z*sl), S2(X*s2,Z*s2), Tl(X*tl, 
Z*tl) and T2(X*t2,Z*t2) are given by coordinates of qsl 
(X f sl,f) and qs2(X , s2,f), qtl(X'tl,f) and qt2(X't2,f): 



30 



35 



Equations (26) and (27) are similarly useful to equations 
(23) and (24) to lead to equation (25). 

Simulation for Testifying the Accuracy 

The accuracy of detecting the position or attitude accord- 
ing to the principle of the present invention is testified by 
means of a simulation. 

In the simulation, a rectangular plane of 100 inch size 
(1500 mmx2000 mm) is given, the four corners being the 
characteristic points, and the target point at the center of the 
rectangular plane, i.e., m-1, n-1. The attitude of the rect- 
angular relative to the image sensing plane is given by 
angles, y=5° and <J>=— 60°. And, the distance between the 
target point and the center of the image sensing plane is 2000 
mm. On the other hand, the distance f is 5 mm. 

According to the above model, the coordinates of char- 
acteristic points on the image sensing plane are calculated by 
means of the ordinary perspective projection matrix. And the 
resultant coordinates are the base of the simulation. 

The following table shows the result of the simulation, in 
which the attitude parameter, angles y and <|> and the position 
parameter, ratio m and n are calculated by means of the 
equations according to the principle of the present invention. 
The values in the table prove the accuracy of the attitude and 
position detection according to the present invention. 



(24) 45 



50 



Equation (23) is given by the X'-coordinate of vanishing 
characteristic points qsl(X'sl) and qs2(X's2), distance f and 
angle <(>, while equation (24) by the X'-coordinate of van- 
ishing characteristic points qtl(X f tl), qt2(X't2), distance f 
and angle <(). With respect to angle tan $ is given by 
equation (18). 

FIGS. 18A and 18B represent conversion from ratio m 55 
and ratio n to a coordinate of target point Ps in which 
characteristic point Q3 is set as the origin of the coordinate. 
In more detail, FIG. 18B is shown in accordance with X*-Y* 
coordinate in which Om (0,f), which is in coincidence with 
target point Ps, is set as the origin, while FIG. 18A shown in 
accordance with U-V coordinate in which a point corre- 
sponding to characteristic point Q3 is set as origin O. 
Further, characteristic points 02 and Q4 in FIG. 18B cor- 
respond to Umax on U-axis and Vmax on V-axis, 
respectively, in FIG. 18A, According to FIG. 18A, coordi- 
nate of target point Ps (u,v) is given by the following 
equation (25): 



ATTITUDE OF GIVEN RECTANGULAR PLANE RELATIVE 
TO IMAGE SENSING PLANE vs. POSITION OF TARGET 
ON THE GIVEN RECTANGULAR PLANE 

Case of Rotation Case of Rotation 

of Vanishing of Vanishing 

line S1S2 by Angle a for line T1T2 by Angle p for 

Coincidence with X-axis Coincidence with Y-axis 



60 



65 
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-59.967 
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tan y 
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5.044 


4.988 


Position 
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0.996 


0.996 


on 


(- OmSl/ 
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OmS2) 






plane 
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1.000 


1.000 


(= OmTl/ 
OmT2) 







Embodiment 2 

FIG. 19 represents a perspective view of Embodiment 2 
according to the present invention. Embodiment 2 especially 
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relates to a pointing device for controlling the cursor move- 
ment or the command execution on the image display of a 
personal computer FIG. 19 shows the system concept of the 
pointing device and its method. 

Embodiment 2 corresponds to a system of Graphical User 5 
Interface, while Embodiment 1 to a position detector, which 
realizes the basic function of Graphical User Interface. Thus, 
the function of Embodiment 1 is quite suitable to Graphical 
User Interface according to the present invention. The basic 
principle of Embodiment 1, however, is not only useful in 10 
Graphical User Interface, but also in detecting the position 
of a target point or the attitude of a three-dimensional object 
in general. 

Further, the information of attitude detection according to 
the present invention is utilized by the position detection, 15 
and then by Graphical User Interface. In terms of the 
position detection or Graphical User Interface, the target 
point should be aimed for detection. However, any specific 
target point need not be aimed in the case of solely detecting 
the attitude of an object as long as the image of necessary 20 
characteristic points of the object are formed on the image 
sensing plane. 

Referring back to Embodiment 2 in FIG. 19, 100 is a main 
body of the pointing device, 110 a given screen plane, 120 ^ 
a personal computer (hereinafter referred to as PC), 121 a 
signal receiver for PC 120, and 130 a projector. Projector 
130 projects display image 111 on screen plane 110. The four 
corners Ql, Q2, Q3 and Q4 of display image 111 are the 
characteristic points, which define the shape of display 3Q 
image 111, the shape being a rectangular. Main body 100 is 
for detecting coordinates of a target point Ps on screen plane 
110 toward which cursor 150 is to be moved. An operator in 
any desired place relative to screen 110 can handle main 
body 100. Broken line 101 is the optical axis of the image 35 
sensing plane of camera 1 (not shown) located inside main 
body 100, broken line 101 leading from the center of the 
image sensing plane perpendicularly thereto to target point 
Ps on screen plane 110. 

According to the Embodiment, the characteristic points 40 
correspond to the four corners of display image 111 pro- 
jected on screen plane 110. However, the characteristic 
points may exist at any locations within screen plane 110. 
For example, some points of a geometric shape within 
display image 111 projected on screen plane 110 may act as 45 
the characteristic points. Alternatively, specially prepared 
characteristic points may be projected within screen plane 
110. The characteristic points may not be independent 
points, but may be the intersection of two pairs of parallel 
lines which are perpendicular to each other. Further, the 50 
characteristic points may not be the projected images, but 
may be light emitting diodes prepared on screen plane 110 
in the vicinity of the display image 111. 

FIGS. 20 and 21 represent the block diagram and the 
perspective view of the system concept for main body 100, 55 
respectively. The configuration of main body 100 is basi- 
cally same as Embodiment 1 in FIG. 2. However, left-click 
button 14 and right-click button 15 are added in FIGS. 20 
and 21. Further, image processor 5 of FIG.2 is incorporated 
in PC 120 in the case of FIGS. 20 and 21. Thus, main body 60 
100 is provided with output signal processor 9 for transfer- 
ring the image data to PC 120 by way of signal receiver 121. 

The functions of left-click button 14 and right-click 
button 15 are similar to those of ordinary mouse, respec- 
tively. For example, left -click button 14 is single-clicked or 65 
double-clicked with the cursor at an object such as an icon, 
a graphics or a text to execute the command related to the 
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object. A click of right-click button 15 causes PC 120 to 
display pop-up menu at the cursor position, just as the 
right-button click of the ordinary mouse does. The move- 
ment of the cursor is controlled by shutter release switch 8. 

Now the block diagram of PC 120 to which the image data 
or command execution signal is transferred from the main 
body 100 will be described in conjunction with FIG. 22. The 
detailed explanation of image processor 5, which has been 
done in Embodiment 1, will be omitted. 

PC 120 receives signals from main body 100 at signal 
receiver 121. Display 122 and projector 130 are connected 
to PC 120. Display 122, however, is not necessarily required 
in this case. 

Image data transferred by main body 100 is processed by 
image processor 5 and is output to CPU 123 as the coordi- 
nate data of the target point. Cursor controller 124 which 
corresponds to an ordinary mouse driver controls the motion 
of the cursor. Cursor controller 124 consists of cursor 
position controller 125, which converts the coordinate data 
of the target point into a cursor position signal in the PC 
display system, and cursor image display controller 126, 
which controls the shape or color of the cursor. Cursor 
controller 124 may be practically an application program or 
a part of the OS (operating system) of PC 120. 

Now, the operations of the pointing device of this embodi- 
ment will be described in detail. Basic operations of the 
pointing device are the position control of cursor and the 
command execution with the cursor at a desired position. 
The function of the pointing device will be described with 
respect to these basic operations. 

FIGS. 23, 24 and 25 represent flow charts for the above 
mentioned operations of the pointing device according to the 
present invention. 

The operation of the pointing device for indicating the 
target point on the screen and moving the cursor toward the 
target point is caused by a dual-step button which controls 
both the shutter release switch 8 and the laser power switch 
7. This operation corresponds to that of the ordinary mouse 
for moving the cursor toward a desired position. 

In step S200 in FIG. 23, the power source of laser power 
switch 7 is turned on by a depression of the dual-step button 
to the first step, which causes the emission of the infrared 
laser beam. The operator may point the beam at any desired 
point on the screen in the similar manner to that in the 
ordinary laser pointer. 

In step S201, the operator aims the laser beam at the target 
point. Instep S202, the dual-step button is depressed to the 
second-step to make the laser power switch 7 off. In step 
S203, it is discriminated whether the mode is an one-shot 
mode or a continuous mode according to the time duration 
of depressing the dual-step button to the second step. In this 
embodiment, a threshold of the time duration is set to two 
seconds. However, the threshold may be set at any desired 
duration. If the dual-step button is kept in the second step 
more than two seconds, it is discriminated that the mode is 
the continuous mode to select step S205. On the other hand, 
if the time duration is less than two seconds in step S203, the 
one-shot mode in S204 is selected. 

In the case of the one-shot mode, shutter release switch 8 
is turned on to take the image in step S206 of FIG. 24. As 
steps S207 to S210 in FIG. 24 are the same as the basic flow 
chart in FIG. 7 described before, the explanation is omitted. 

In step S211, the coordinate of the target point is con- 
verted into a cursor position signal in the PC display system 
and transferred to projector 130 by way of CPU 123 and 
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display drive 127. In step S212, projector 130 superimposes In summary referring back to FIG. 22, the image data 

the cursor image on the display image 111 at the target point taken by the main body 100 with the target point aimed with 

Ps. the laser beam is transmitted to PC 120. In PC 120, the 

In this stage, the operator should decide whether or not to image data is received at signal receiver 121 and transferred 

push left-click button 14. In other words, the operator would 5 to input/output interface (not shown), which processes the 

be willing to click the button if the cursor is successfully image data and transfers the result to image processor 5. 

moved to the target point. On the other hand, if the desired with the image processor 5, characteristic points are 

cursor movement is failed, which would be caused by the detected on the basis of the image data to calculate their 

depression of the dual-step button with the laser beam at an coordinates. The coordinates of the characteristic points are 

incorrect position, the operator will not click the button. Step 10 procesS ed t0 calculate the coordinate of the target point, 

213 is for waiting for this decision by the operator. In the which ^ transferred t0 cursor position con troller 125 to 

case of failure m moving the cursor to the desired position, mQve ^ cpu 123 contfols those ^ons. ^ 

the operator will depress the dual-step button again, and the . ... , . t P 4U * „ ♦ 

flow will jump back to step S200 in FIG. 23 to restart the resultant cursor Position corresponds to that of the target 

function. On the contrary, if the operator pushes left-click P oint ° n the ^ d^i^ ^ *™ te * c C ° mmand f 

button 14, the flow is advanced to step S214 to execute the 15 execution signal to PC 120 with reference to the position of 

command relating to the object on which the cursor is lne cursor - 

positioned. The command execution signal caused by the In more detail, position controller converts the informa- 

left-click button 14 of main body 100 is transmitted to PC tion of the target point given in accordance with the X*-Y* 

120. Step S215 is the end of the command execution. coordinate as in FIG. 18B into a cursor controlling infor- 

In step S212, the cursor on the screen stands still after 20 mation S iven in accordance with the U-V coordinate as in 
being moved to the target point. In other words, cursor F *G. Im- 
position controller 125 keeps the once-determined cursor CPU 123 activates a cursor control driver in response to 
position signal unless shutter release switch 8 is turned on an interruption signal at input/output interface to transmit a 
again in the one-shot mode. Therefore, left-click button 14 cursor control signal to display drive 127. Such a cursor 
can be pushed to execute the command independently of the 25 control signal is transferred from display drive 127 to 
orientation of main body 100 itself if the cursor has been projector 130 to superimpose the image of cursor on the 
successfully moved to the desired position. display image. 

FIG. 25 is a flow chart for explaining the continuous On the other hand, the command execution signal trans- 
mode, which corresponds to the details of step S205 in FIG. mitted from main body executes the command depending on 
23. In step 221, shutter release switch 8 is turned on to take 30 the position of the cursor in accordance with the OS or an 
the image. In the continuous mode, a train of clock pulses application of CPU. 

generated by the controller 4 at a predetermined interval The small and light wait pointing device according to the 

governs shutter release switch 8. This interval stands for the present invention needs not any mouse pad or the like as in 

interval of a step-by-step movement of the cursor in the 35 the ordinary mouse, but can be operated in a free space, 

continuous mode. In other words, the shorter the interval is, which greatly increases a freedom of operation. Besides, an 

the smoother is the continuous movement of the cursor. The easy remote control of PC is possible with a desired object 

set up of this interval is possible even during the operation image on a screen plane pointed by an operator himself, 

by a set up dial or the like (not shown). Further, the pointing device according to the present 

Shutter release button 8 is once turned on at every pulse 4Q invention may be applied to a gun of a shooting game in such 

in the train of the clock pulses in Step 221. And, Steps 221 a manner that a target in an image projected on a wide screen 

to 227 are completed prior to the next pulse generated, and is to be aimed and shot by the pointing device as a gun. 

the next pulse is waited for in Step 221. Thus, Steps 221 to Now, a coordinate detection of a target point in an image 

227 are cyclically repeated at the predetermined interval to projected with a distortion on a screen plane will be 

cause the continuous movement of the cursor. 45 described. 

The repetition of Steps 221 to 227 continues until the In a case of projecting an original of true rectangle to a 

dual-step button is made off the second step. Step 227 is for screen plane with the optical axis of the projector perpen- 

terminating the continuous mode with the dual-step button dicular to the screen plane, the projected image would also 

made off the second step. be of a true rectangle provided that the optical system of the 

According to a modification of the embodiment, the 50 projector is free from aberrations, 

cyclic repetition of Steps 221 to 227 may be automatically On the contrary, if the original of true rectangle is 

controlled depending on the orientation of the main body projected on the screen plane inclined with respect to the 

100. In other words, the train of clock pulses is intermitted optical axis of the projector, an image of a distorted quad- 

not to turn on shutter release button 8 in step S221 when the rangle would be caused on the screen plane. The main body 

laser beam is outside the area of display image 111, and is 55 takes the distorted quadrangle on the screen plane with the 

generated again when the laser beam comes back inside the optical axis of the main body inclined with respect to the 

area. screen plane to cause a further distorted quadrangle on the 

Although not shown in the flow in FIG. 25, left-click image sensing plane of the main body, 

button 14 may be pushed at any desired time to execute a According to the principle of the present invention, 

desired command. Further, if main body 100 is moved with 60 however, the calculations are made on the assumption that 

left-click button 14 kept depressed along with dual -step the image on the screen plane is of a true rectangle. This 

button depressed to the second step, an object in display means that the distorted quadrangle on the image sensing 

image 111 can be dragged along with the movement of the plane is considered to be solely caused by the inclination of 

cursor. the screen plane with respect to optical axis of the main 

As described above, the cursor control and the command 65 body. In other words, the calculated values do not represent 

execution of the embodiment according to the present inven- actual parameters of the attitude of the screen plane on 

tion can be conveniently practiced as in the ordinary mouse. which the first mentioned distorted quadrangle is projected. 
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But, an imaginary parameters are calculated according to an 
interpretation that the final distorted quadrangle on the 
image sensing plane would be solely caused by the attitude 
of the screen plane on which a true rectangle is projected. 

More precisely, the main body cannot detect at all whether 5 
or not the distorted quadrangle on the image sensing plane 
is influenced by the inclination of the screen plane with 
respect to optical axis of the projector. But, the main body 
carries out the calculation in any case on the interpretation 
that the image projected on the screen plane is of a true 1Q 
rectangle. Ratio m and ratio n for determining the position 
of the target point on the screen plane are calculated on the 
basis of thus calculated attitude. 

According to the present invention, however, it is experi- 
mentally confirmed that ratio m and n calculated in accor- 
dance with the above manner practically represent the 
position of the target on the original image in the projector 
as long as such original image is of a true rectangle. In other 
words, the determination of the target on the original image 
in the projector is free from the inclination of the optical axis 
of the projector with respect to the screen plane, which 20 
inclination would cause a distorted quadrangle on the screen 
plane. Therefore, a correct click or drag in the case of the 
graphic user interface of a computer or a correct shot of a 
target in the case of shooting game is attained freely from a 
possible distortion of the image projected on a wide screen. 25 

What is claimed is: 

1. A graphical user interface for generating an interface 
signal with an aimed-at target on a display comprising: 

an image sensor having an image plane on which an 
image including characteristic points is formed, a pre- 30 
determined position of the image plane corresponding 
to a target point; 

a processor for calculating the position of the target on the 
display using parameters of an attitude of the display 
relative to the image plane on the basis of four char- 35 
acteristic points; and 

a signal generator for generating an interface signal with 
the position of the target calculated. 

2. The graphical user interface according to claim 1, 
further comprising an aiming device for helping the image 40 
sensor to form the image of the display with the target in 
coincidence with the predetermined position. 

3. The graphical user interface according to claim 1, 
further comprising an aiming device including a light beam 
emitter for emitting a light beam along the optical axis of the 45 
image sensor toward the display, whereby the image of the 
target is in coincidence with the predetermined position on 
the image plane when the light beam hits the target on the 
display. 

4. The graphical user interface according to claim 1, 50 
further comprising an aiming device including a viewfinder 
for aiming the target. 

5. The graphical user interface according to claim 4, 
wherein the viewfinder includes an optical finder for opti- 
cally aiming the target. 55 

6. The graphical user interface according to claim 5, 
wherein the viewfinder includes a monitor of the output of 
the image sensor. 

7. The graphical user interface according to claim 1, 
further comprising a controller for ordering the image sensor 60 
to take the image of the display. 

8. The graphical user interface according to claim 7, 
wherein the controller is designed to order the image sensor 
to take one image in response to a manual operation. 

9. The graphical user interface according to claim 7, 65 
wherein the controller is designed to order the image sensor 

to take a series of images in succession. 
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10. The graphical user interface according to claim 1, 
wherein the processor includes an image processor for 
identifying the positions of the characteristic points on the 
image plane, a first processor for calculating a first data on 
the basis of the positions of the characteristic points on the 
image plane, and a second processor for calculating a second 
data on the basis of the first data and the positions of the 
characteristic points on the image plane, the position of the 
target on the display being detected on the basis of the 
second data. 

11. A position detector for detecting a position of a target 
point on a given plane having a plurality of characteristic 
points comprising: 

an image sensor having an image plane on which an 
image of the given plane is formed with the character- 
istic points included in the image, a point of the image 
which is formed at a predetermined position of the 
image plane corresponding to the target point to be 
detected on the given plane; 

an image processor for identifying the positions of the 
characteristic points on the image plane; 

a first processor for calculating a first data on the basis of 
the positions of the characteristic points on the image 
plane; and 

a second processor for calculating a second data on the 
basis of the first data and the positions of the charac- 
teristic points on the image plane, the position of the 
target point on a given plane being detected on the basis 
of the second data. 

12. The position detector according to claim 11, wherein 
the first data includes information of attitude of the given 
plane relative to the image plane. 

13. The position detector according to claim 11, wherein 
the first processor calculates the first data in accordance with 
the three-dimensional perspective projection. 

14. The position detector according to claim 13, wherein 
the first processor includes a calculator for calculating a pair 
of vanishing points on the basis of the positions of the 
characteristic points on the image plane. 

15. The position detector according to claim 14, wherein 
the first processor rotates the positions of the characteristic 
points around an axis perpendicular to the image plane so 
that one of the vanishing points comes on one of the 
coordinate axes on the image plane. 

16. The position detector according to claim 15, wherein 
the second processor includes a first line calculator for 
calculating a first line between the predetermined position 
and one of the vanishing point, a second line calculator for 
calculating a second line between a pair of characteristic 
points and a intersection calculator for calculating an inter- 
section between the first and second lines. 

17. The position detector according to claim 11, wherein 
the first processor includes a calculator for calculating a pair 
of vanishing points on the basis of the positions of the 
characteristic points on the image plane, and wherein the 
second processor includes a first line calculator for calcu- 
lating a first line between the predetermined position and 
one of the vanishing point, a second line calculator for 
calculating a second line between the position of a pair of 
characteristic points and a intersection calculator for calcu- 
lating an intersection between the first and second lines. 

18. The position detector according to claim 11, further 
comprising an aiming device for helping the image sensor to 
form the image of the given plane with the target point in 
coincidence with the predetermined position. 

19. The position detector according to claim 18, wherein 
the aiming device includes a light beam emitter for emitting 
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a light beam along the optical axis of the image sensor 
toward the given plane, whereby the image of the target 
point is in coincidence with the predetermined position on 
the image plane when the light beam hits the target point on 
the given plane. 5 

20. The position detector according to claim 18, wherein 
the aiming device includes a viewfinder for aiming the target 
point. 

21. An attitude detector for detecting an attitude of a given 
plane having a plurality of characteristic points comprising: 10 

an image sensor having an image plane on which an 
image is formed with four characteristic points 
included in the image, a predetermined position being 
the origin of the coordinates of the image plane; 

a helping device for aiming the given plane; 15 

an image processor for identifying the positions of four 
characteristic points on the image plane; 

a first processor for calculating a pair of vanishing points 
on the basis of the positions of four characteristic points 2 o 
on the image plane; and 

a second processor for calculating intersection points on 
the basis of the vanishing points and the positions of the 
four characteristic points on the image plane, the atti- 
tude of the given plane being detected on the basis of 25 
the intersection points. 

22. The attitude detector according to claim 21, wherein 
the first processor rotates the positions of the characteristic 
points around an axis perpendicular to the image plane so 
that one of the vanishing points comes on one of the 30 
coordinate axes on the image plane. 

23. The attitude detector according to claim 21, wherein 
the second processor includes a first line calculator for 
calculating a first line between the predetermined point and 
one of the vanishing points, a second line calculator for 35 
calculating a second line between a pair of characteristic 
points and a intersection calculator for calculating an inter- 
section between the first and second lines. 

24. The attitude detector according to claim 21, wherein 
the first processor includes a calculator for calculating a pair 40 
of vanishing points on the basis of the positions of the 
characteristic points on the image plane, and wherein the 
second processor includes a first line calculator for calcu- 



,885 Bl 

22 

lating a first line between the predetermined point and one 
of the vanishing points, a second line calculator for calcu- 
lating a second line between a pair of characteristic points 
and a intersection calculator for calculating an intersection 
between the first and second lines. 

25. The attitude detector according to claim 21, wherein 
the helping device includes a light beam emitter for emitting 
a light beam along the optical axis of the image sensor 
toward the given plane. 

26. The attitude detector according to claim 21, wherein 
the helping device includes a viewfinder with a cross being 
optically equivalent to the predetermined position. 

27. A graphical user interface for generating an interface 
signal with an aimed-at target on a display comprising: 

an image sensor having an image plane on which an 
image including characteristic points is formed, a pre- 
determined position of the image plane corresponding 
to the target; 

an aiming device for helping the image sensor to form the 
image of the display with the target in coincidence with 
the predetermined position; 

a transmitter for transmitting the output of the image 
sensor to an outside processor for a calculation of the 
position of the target on the display relative to four 
characteristic points; and 

a signal generator for generating an interface signal with 
the position of the target calculated. 

28. The graphical user interface according to claim 27, 
wherein the aiming device includes a light beam emitter for 
emitting a light beam along the optical axis of the image 
sensor toward the display, whereby the image of the target 
is in coincidence with the predetermined position on the 
image plane when the light beam hits the target on the 
display. 

29. The graphical user interface according to claim 27, 
wherein the aiming device includes a viewfinder for aiming 
the target. 

30. The graphical user interface according to claim 27, the 
graphical user interface is for use in a computer. 

31. The graphical user interface according to claim 27, the 
graphical user interface is for use in a shooting game. 

* * * * * 
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